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FOREWORD 



This paper is one of several prepared by consultants to The Project on Adult English as a Second 
Language (ESL) of the Southport Institute for Policy Analysis. These papers were commissioned to help 
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present state and future directions of ESL service in the United States. 
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As a result, the views expressed u\ this paper are those of the author alone. They do not 
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involved with the Project on ESL. 
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Introduction 

The words quality, standards and accountability are hard to avoid in any 
discussion of adult ESL today. Stakeholders: administrative and instructional staff, 
learners, funders and the community, including the business community, want to know 
what they are getting for their investments. Thus, program administrators are being 
asked to reveal what they are trying to achieve, how good of a job they are doing and 
how they can prove it. These are not unreasonable requests, bat program people at all 
levels of implementation are finding it difficult to determine what constitutes quality, 
what are acceptable standards of performance, and what is required to demonstrate 
accountability for adult ESL instruction. 

Difficulty stems from the nature of adult education in general, how it is funded, 
and the tendency to gloss over the real issues of teaching adult immigrants English. 
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Nevertheless, the National Literacy Act of 1991 requires that state and local programs 
receiving funds from the Adult Education Act (AEA) will establish indicators of program 
quality to help judge the success of adult education. A preliminary examination of the 
efforts of the federal and state governments in carrying out the requirements of the 
National Literacy Act bears the news that while all states and the federal government 
have submitted indicators of program quality, progress in using them to demonstrate 
accountability is slow and efforts to establish standards of success are minimal. The 
purpose of this paper is to take a look at exactly what has been accomplished by the 
states, why it has been so difficult and what policy recommendations can be made to 
establish serious systems of accountability. 

Adult education encompasses a spectrum of services designed to improve the 
acquisition of language, literacy and basic skills for aduUs who have not had the 
opportunity to develop such skills previously. This publication focuses specifically on 
issues that relate to ESL learners because it is an area of adult education that is 
frequently overlooked, forgotten or not deemed to be a priority by the field and funders 
(Wrigley & Guth, 1992). Specifically, the Southport Institute for Policy Analysis 
estimates that there are 11 to 14 million adults who are limited in their English 
proficiency and may be in need of second language education. Of those, only about 1.2 
million students are served aduU basic education services at a cost of about 64 million 
dollars (Chisman, Wrigley, and Ewen, 1993). Obviously, the little funding that is 
available for this population must be used as prudentiy and effectively as possible. 



ERIC 



This is not to say that no steps have been taken to demonstrate accountability in 
adult education. The National Literacy Act of 1991, legislated that adult education 
programs funded under the Adult Education Act identify indicators of quality that would 
be used to demonstrate success. For almost two years, the federal government and the 
states have been involved in an effort to define quality in adult education. 

Thus far, the federal government and the states have all developed documents 
which outline quality indicators. Indicators have been developed with varying degrees 
of input from the field (Center on Budget and Policy Priorities, 1993) and varying 
degrees of enthusiasm. While some reactions to this requirement can best be summarized 
with the expression "it's about time/' others feel that the requirement is unfair given that 
other postsecondary institutions such as community colleges and universities are not 
legislated to demonstrate outcomes or successes, yet they receive far more than 128 
million dollars of federal funds. Still others are concerned about how the indicators will 
be used, and even though performance standards are not required by law, practitioners 
are anxious about the possibility of de 'eloping performance standards for programs that 
serve such diverse populations (The Literacy Assistance Center, 1992). 

Bob Shireman, a legislative aide to Senator Paul Simon of Illinois, one of the 
authors of the original legislation of the National Literacy Act, argues that program 
performance standards are needed to keep funding honest and keep politics out of funding 
decisions ("Shireman: Program Performance," 1993). But others approach the process 
with caution, citing concern for: (1) the lack of adequate resources to implement quality 



programs and meet performance standards, (2) the inappropriateness of ineasures to 
determine if performance standards have been met, and (3) the identification of 
inappropriate standards for the population that is served (•'Shireman: Program 
Performance," 1993). 

Regardless of the feelings that permeate this movement, all states met the 
requirement of submitting indicators to the U.S. Department of Education (USDOE) in 
July of 1993 and many are in the process of refining or expanding their documents. In 
addition, the National Institute for Literacy (NIL) recently completed a Request for 
Proposals process "to help state and local leaders create performance measurement, 
reporting and improvement systems that will enhance states' abilities to monitor the 
results of their adult literacy systems and to make continuous improvements in their 
literacy programs and policies" (Federal Register, 1993, p. 41940). Unfortunately, only 
five states are funded to create systems, with only two having significant ESL populations 
(New York and Hawaii). 

There are five major problems, however, in what has been accomplished to date. 
The problems are: 

1) The indicators of quality that have been established are inconsistent across 
states and are not comprehensively reflective of what constitutes quality 
in adult education. 
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2) The federal government and the states have avoided identifying what they 
are exactly trying to achieve in programs and specific learner outcomes 
have been left unstated. 

3) There is little indication of what type of evaluation should accompany the 
review of indicators, little discussion of what constitutes quality evaluation 
in adult education, and little attempt to evaluate the quality indicators that 
have been developed to date. 

4) There is even less attention to the systems of monitoring and compliance 
that will be utilized to see if programs reflect the established indicators of 
quality and almost no indication of what will happen to programs if they 
do not demonstrate accountability. 

5) Minimal attention is paid to ESL instruction in the work that has been 
done so far. 

In short, the efforts of the federal government, the states and the NIL do not 
separately or collectively provide the necessary leadership to unite adult educators to 
thoroughly study the issue of accountability; to assess what needs to be done about it so 
that programs can demonstrate that they are making a difference and getting the most 
"bang for the buck;" and to mobilize the field so that local, state and federal efforts are 
organized to accomplish the awesome ta ,k of developing quality programs that meet 
standards and demonstrate accountability. 



To overcome these problems it is necessary to: (1) examine the strengths and 
weaknesses of accountability schema that have been developed in other educational 
settings, (2) define what is "state of the art" in terms of quality for adult ESL, (3) 
describe the components of a system for accountability that would be appropriate for 
adult ESL programs, (4) assess the status of federal and state initiatives in establishing 
systems of accountability for adult ESL education to date, (5) identify gaps in the present 
accountability systems that have been developed for adult ESL programs and (6) make 
recommendations for policy makers that will enable federal and state governments and 
local providers to develop appropriate models of accountability for adult ESL programs. 

Accountability Schema: What We Know About Other Programs 

Accountability is a concept which involves determining the effects of educational 
programs and institutions by examining the outcomes in relation to the use of resources 
(Dressel, 1978). Various stakeholders such as administrative and instructional staff, 
learners, funders and the community, including the business community, are interested 
in how educational programs demonstrate accountability. While many feel that 
accountability has never been more important to the institution of education than today, 
it is both a pervasive and elusive concept that is difficult to realize because inherent in 
accountability is defining what constitutes quality and standards. 

Quality is an overarching goal of programs that is difficult to define but often 
recognized when observed. Key to the notion of quality is the idea that program 



providers have common elements that exist in good programs, some essential 
components, and ways to measure the quality that is evidenced in the essential 
components. 

Bogue and Saunders (1992) see quality as "conformance to mission specifications and 
goal achievement-within publicly accepted standards of accountability and integrity" (p. 
20). They further suggest that while measuring quality is complicated, it must involve: 

1. selecting evidences and indicators of performance, 

2. acquiring and analyzing data, 

3. setting an appropriate standard of performance, 

4. identifying who will make the judgments of performance and how. (p. 17) 

Establishing standards involves defining what quality is and setting the parameters 
of what makes a program effective or ineffective. Ideally, standards are based on 
concrete data or information about what works and what does not work and what has 
been achieved or not achieved. Important to the development of standards is the 
identification of performance measurement, or some yardstick which enables program 
providers to determine how well they have met or reached a standard. There are three 
possible choices for developing standards according to Bogue and Saunders (1992, p. 5): 
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1 . A criterion standard in which performance is compared to a predetermined 
criterion level. 

2. A normative or comparative standard, in which performance is judged 
against the performance of another program, person or group of persons. 

3. A connoisseuring standard in which performance is judged against the 
opinions and values of a panel of judges. 

Quality, standards and accountability are interrelated - one cannot exist without 
the other. Ideally, they function synchronously to produce programs that meet the needs 
and goals of learners as efficiently and effectively as possible. In essence, accountability 
is the ultimate outcome of a quality program that is able to demonstrate that standards 
have been met through systems which measure program and learner outcomes. But even 
though quality, standards and accountability are on the agendas of many educational 
programs and legislation exists to require most programs that receive federal and state 
dollars to develop accountability systems, no area of education has been able to construct 
an adequate approach. 

There are several significant reasons why it is so difficult to create accountability 
systems. First, substantial resources are required to develop appropriate accountability 
systems. Because of inadequate funding for education in general, most educational 
resources are required for serving students. This problem is severe for adult education 
since it is grossly underfunded in comparison to primary, secondary, higher and 
vocational education and employment training programs. Under Adult Basic Education 



grant approximately, 95% of the adult education resources are earmarked for serving 
students leaving only 5% for administration. Therefore, most adult education programs 
cannot afford to set up systems of accountability. 

Second, while quality might be recognizable when it is observed, the process of 
establishing standards and systems of measurement is a "thorny" issue for educators, 
especially adult ESL educators. Difficult questions emerge such as can performance 
standards truly reflect program quality when quality is so difficult to define; can program 
standards be developed to examine the performance of diverse adult populations who 
come to educational programs with a variety of needs, backgrounds and experience; and 
what are the most important outcomes for programs and students and can those outcomes 
be accurately measured given the inadequacy 3f existing assessment tools. These 
questions, and a myriad of others, must be addressed as part of the process of 
establishing accountability. 

Third, there is a tension between product and process in measuring the outcomes 
of euucation. On the one hand, a movement to establish national standards that are 
measured with national tests is pre^'alent throughout education (see Bogue & Saunders 
1992; National Council on Education Standards and Testing, 1992). The view that the 
results of education can be quantified and measured according to a "one size fits all" 
system is contrary to the emerging educational theories that promote emancipatory 
practice, participatory learriing and holistic instruction (see Fingeret & Jurmo, 1089; 
Kerfoot, 1993; Wrigley and Guth, 1992). 
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Finally, for adult education, the challenge of establishing systems of accountability 
is especially overwhelming given the status of a workforce that is largely part-time and 
not part of an organized profession of adult educators. In order to create and implement 
systems of accountability, large scale professional development is required. 

Nevertheless, the requirement to demonstrate accountability in adult education is 
real. Responding thoughtfully aiid carefully is crucial. Further, examining the 
approaches to accountability of other educational areas can help adult educators avoid 
unnecessary pitfalls. 

Programs which receive funding from the Carl D. Perkins Vocational and Applied 
Technology Education Act of 1990 have been mandated through legislation to measure 
student gains in academic skills, and occupational performance (Section 115(a) and (b). 
In addition, this legislation, encourages state ; to serve special populations effectively 
(Section 118(a). Accordingly, each state must develop performance standards that 
measure: 1) learning gains, 2) competency or job skill attainment, 3) placement into 
employment or other training programs, and 4) retention in or completion of secondary 
school or its equivalent (CouJelli & Kutner, 1992). 

Technical assistance for Carl Pe'kins funded programs to meet the regulations is 
provided through the National Center for Research In Vocational Education (NCRVE). 
The Center advocates an outcomes assessment approach to accountability which relies 
greatly on the collection of information for improving the program and determining the 
impact of institutional and program efforts on student achievement (NCRVE, 1992). 
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This approach is seen as critical to providing quality programs for dealing with the 
diversity of students in postseccndary programs. Specific recommendations for 
developing outcomes assessment system according to Bragg and Harmon (1992), are: 

1. Focused and realistic mission statement aAd goals should be tied to 
outcomes assessment and institutional effectiveness processes. 

2. Issues of cultural diversity and pluralism should be adequately addressed 
in the system. 

3. The system should reflect top down support and bottom up devdopnient 

coupled with adequate resources. 

4. The needs of the customer should be considered a top priority. 

5. Existing information should be audited and used to learn about how the 
internal and external processes of an organization are functioning, 
reinforced by and coordinated with essential data collection activities. 

6. Longitudinal databases using qualitative and qi^titative data on student 
performance should be created. 

7. Evaluation and assessment information should be used for improving and 
plamiing programs. 

In addition, Hoachlander (1991) suggests that a good accountability system 
includes a clearly defined system of measures and standards, which is manageable and 
based on accurate, timely and uncomplicated data collection. 
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The approach to establish accountability systems taken by vocational education 
through the work of NCRVE is valuable in terms of the emphasis on setting standards 
that are based on viable data collection and developing systems of data collection that 
provide useful information for program planning and improvement. The need to strike 
a balance between spending time to create effective programs and demonstrating that they 
are effective is supported as well. Valuable technical assistance through NCRVE is 
apparent. But vocational education's overall approach is weak in terms of monitoring 
components and systems for defining quality in vocational education programs. 

While the original intent of the Job Training Partnership Act (JTPA) was to 
allocate funds to provide training and jobs for disadvantaged adults and out of school 
youth, the actual implementation of job training for this population required education 
to build basic skills needed for training and employment. Since its inception, the JTPA 
program has required programs to identify standards which specify acceptable levels of 
performance and indicators of program quality. Legislation requires that programs 
measure employment, earnings and reduction of welfare dependency for participants. 
There are four core performance standards adopted by the U.S. Department of Labor 
(USDOL) for JTPA, Title II A programs: 

1 . employment rate at follow-up 

2. weekly earnings at follow-up 

3. youth entered employment rate 

4. youth employ ability rate 
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Follow-up occurs three months after program termination and numeric goals for 
the performance measures to monitor program outcomes in each state are set by the 
USDOL. These outcomes may be adjusted according to state and local conditions. More 
funding may be obtained when providers exceed standards; and all providers must have 
a management information system for keeping data related to standards, demographics, 
fiscal matters and other program requirements. 

Performance standards for JTPA 123 funds, which are typically awarded to serve 
high risk populations, like language minority students, must demonstrate educational 
gains in specified time periods that are determined locally. 

While JTPA was one of the first programs to establish performance standards, its 
approach was based on setting standards without much regard to the process issues 
involved in establishing quality programs. The need to develop performance standards 
that are based on systematic qualitative and quantitative data collection and research is 
an important lesson to learn regarding the shortcomings of the JTPA approach to 
accountability. A specific lesson learned from JTPA regarding performance standards 
is that they have the potential for operationalizing the law of unintended consequences 
and reducing the level of services without revealing much about quality (see Condelli & 
Kutner, 1992; Stein, 1993). 

Another federal program which recently announced the requirement of 
performance standards is the Job Opportunities and Basic Skills Training (JOBS) Program 
of the U.S. Department of Health and Human Services (USDHS). The U.S. Department 
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of Health and Human Services will issue a first on a series of reports on developing 
performance criteria for the program, but essentially, the Department did not meet the 
October 1993 deadline for completion of program standards. The Department is 
experiencing difficulty m establishing the standards because of the many variables that 
affect the JOBS program that cannot be controlled such as the economic condition of the 
community, the mandatory requirements of the program, and the resistance issues related 
to avoiding participation (Texas Department of Human Services, 1993). 

Leadership for establishing systems of accountability and standards of program 
quality for community based programs has al:o been initiated by the Association for 
Community Based Education (ACBE). Work by Sondra Stein (1993) provides a 
framework for assessing program quality which encourages programs to: 

1. define program goals in relation to community needs; 

2. define outcomes that operationalize these goals; 

3. describe outcomes concretely and comprehensively to facilitate 
measurement; 

4. identify structures and processes the program needs to have in place for 
the outcomes to be achieved; 

5. develop systems of documentation that will enable the programs to 
measure change along key parameters, (p. 2) 

While the ACBE framework is strong in terms of balancing product and process, 
it falls short in terms of addressing issues of monitoring effectiveness. For community 
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based programs which have diverse governance structures, there needs to be some system 

of accreditation to ensure quality across programs. 

Higher education addresses accountability in a many ways, but formally through 

the six regional accreditation agencies which are members of the Council on 

Postsecondary Accreditation (COPA) (see Bragg & Harmon, 1992; Bogue & Saunders, 
1992). Outcomes assessment is required to some extent by all six agencies (Marchese, 
1990) and by 1988 the accrediting associations "required evidence of achievement as a 
part of other institutional or programmatic reviews" (p. 371). Regional approaches vary, 
but several accreditation agencies (North Central, Southern, and Western) use an 
effectiveness indicator-based approach that reflects ihe institution's mission statement 
(Bragg & Harmon, 1992). 

In their book. Evidence for Quality . Bogue and Saunders, (1992) review six basic 
practices which reflect an attempt to offer quality programs in higher education: 

• accreditation: the test of goal achievement and improvement 

• rankings and ratings: the test of reputation 

• outcomes: the test of results 

• licensure: the test of professional standards 

• program reviews: the test of peer review 

• follow-up studies: the test of client satisfaction (p. 22). 
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The vastness of programs in higher education and the variety of public and private 
funding sources make it difficult for higher education to approach accountability as a 
whole. Accreditation and professional standards are longstanding efforts to address 
accountability issues in higher education and can be used as models for other areas of 
education. But the collection of data to determine the outcomes and approaching 
accountability systematically are shortcomings of higher education. 

Although states have been developing standards for K-12 for years, a federal 
approach to accountability is being undertaken by the National Council on Education 
Standards and Testing. In a report from the National Council (1992), it concludes tiiat 
national standards should be tied to assessment so that resources can be better targeted 
and higher expectations for students can be created. According to the Council, "student 
standards include specification of the content-what studems should know and be able to 
do--and the level of performance that students are expected to attain" (p.3). Further, the 
Council advocates the development of delivery standards and system performance 
standards so that students are not solely responsible for attaining standards. 
Characteristics of standards, according to the Council, should: 

reflect high expectations, not minimal levels of competency 
provide focus and direction, not become a national curriculum 
be national, not federal 

be voluntary, not mandated by federal government 
be dynamic, not static (p. 3) 
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In order to determine if standards have been met, new assessments and assessment 
systems must be developed which are: (1) individualized, (2) large scale, (3) 
representative of multiple methods for measuring progress, (4) voluntary and 5) 
developmental. Finally, the Council promotes the use of assessment as the "basis for 
system accountability" (p.4). 

Primary concerns of the Councils' approach is an overreliance on assessment. 
While the Council acknowledges that new systems of assessment must be developed, it 
does not give enough credence to the technical assistance and professional development 
that is required to develop those systems. Further, the Council pays little attention to the 
data collection that is required to develop accountability systems and the research needed 
to establish standards. The National Association of Bilingual Education (NABE) and 
Teachers of English to Speakers of Other Languages (TESOL) have worked on standards 
for bilingual education for K-12 programs. NABE standards are designed to provide 
guidelines for teacher preparation programs and cover areas of: l)institutional resources, 
coordination and commitment; 2) recruitment, advisement and retention of potential 
teachers; 3) bilingual/multicultural course work and curriculum; 4) language proficiency 
in English and non-English and ability to teach in those languages; 5) field work and 
practical experiences in bilingual/multicultural classrooms; and 6) lifelong learning, 
commitment to professional involvement (National Clearinghouse of Bilingual Education 
(NCBE), 1993). TESOL standards consider the improvement and effectiveness of 
educational programs fo^ language minority students in public schools. TESOL*s 
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document includes fifteen items that are grouped in four categories related to access 
issues such as access to a positive learning environment, access to appropriate 
curriculum, access to full delivery of services, and access to equitable assessment 
(NCBE, 1993). Unfonunately, NABE and TESOL do not significantly address the 
systemic aspects of implementing these standards. 

Accountability: State Of The Art For Adult Education 

For adult basic education (ABE) specifically, accountability frameworks were 
reviewed by Pelavin and Associates as they prepared to help the federal government 
respond to its obligations required by the National Literacy Act in establishing a model 
for quality indicators. In one report, Condelli, Koloski, and Webb (1992) review the 
states' efforts to conceptualize ^nd measure program quality. The report indicates that 
states have five general areas and measures of program quality that are commonly found 
in the monitoring instruments. These areas include: planning (process, goals, and 
relationship to program components), personnel characteristics, program capacity (type 
of programming, instructional and other service providers, sites, and program 
management), curriculum/instructional methodology, and services to the disadvantaged. 

Pelavin and Associates used this information along with the USDOE^s request to 
add indicators for program, process and content, and the requirement of the National 
Literacy Act to establish quality indicators for recruitment, retention and learning gains. 
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Thus, the federal government's contribution to help states develop models forjudging the 
success of programs includes general quality indicators related to: education gains, 
program planning, curriculum and instruction, staff development, support services, 
recruitment and retention (Office of Vocational and Adult Education (OVAE), 1992). 

In another document written to help states develop evaluation frameworks for state 
adult education programs, Sherman and Condelli (1991) provide guidelines for 
establishing performance standards to use in program evaluation. They recommend that 
"The state must first select which indicators it wants to use as standards, obtain baseline 
data for the indicators and then establish a goal level for each indicator." (p.4). In 
addition, the authors suggest that local providers be included in developing the standards 
and that the state establish an ongoing process for determining how well the programs 
are meeting the standards. 

Seven program providers suggest additional elements for establishing quality for 
adult education programs in a series of papers related to issues inherent in implementing 
quality indicators (see Corley, et al., 1992). Williams, one of the authors, discusses the 
importance of focusing on results and having outcomes as an important criterion for 
measuring successes. In the same report, Mansoor who specifically addresses quality 
and ESL, offers that quality is intrinsically related to the issue of accountability and 
therefore systems must attend to planning; defining success in terms of the program and 
the learner; meeting self-defined standards; being responsive to learner needs; and linking 
evaluation results to program improvement, learner progress and performance. Finally, 
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Corley, of the Maryland State Department of Education, responds to the government's 
indicators of quality, by adding that it is vital to show that minimum standards are met 
before quality can be established. 

At a forum sponsored by the National Center on Adult Literacy in June, 1993, 
in Washington, D.C., another practitioner, Jo Ann Weinberger suggests six 
recommendations for developing standards for literacy programs: 

1. A process of inclusion is needed for developing the indicators. 

2. Different types of standards are needed for different types of programs. 

3. There is a need to determine the most important outcomes and utilize 
research to 5how how to best reach those outcomes. 

4. Different types of student assessments must be utilized. 

5. Professional development is crucial for successful implementation of 
standards. 

6. More funds are needed to support the implementation of an effort to 
achieve standards. 

For adult ESL programs, Guth and Wrigley (1992) conducted a two year study 
to "identify effect) e and innovative instructional approaches, methods, and technologies 
used to provide literacy instruction for adult English as a second language literacy 
students" (p.l). They reviewed ten programs in depth which had been selected from 123 
nominations on the basis of the promising practices existent in the program. Using a 
combination of field research, case study and interviews, Wrigley and Guth identified 
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eight components of quality programs. They are: 1) community outreach, 2) needs 
assessment of the learner community, 3) program design, 4) curriculum, approaches and 
methods, 5) initial assessment and progress evaluation, 6) staff development, and 7) 
support servi.ces. In another publication related to their study, Wrigley and Guth (1992) 
conclude that since ESL programs are diverse in focus and try to accomplish a variety 
of goals, establishing standards might require emphasis on processes that ensure quality 
and programs should be held accountable for having structures, plans and evaluation tools 
that promote quality education. Thus, to move the ESL field toward accountability, 
program standards need to be developed for each of these components. 

In reviewing accountability schema for various educational programs and the 
literature regarding quality characteristics for adult education, there appears to be five 
core components of demonstrating accountability. Each component is described below 
plus critical elements inherent in each component. 

CORE COMPONENT 1. Quality program indicators should be ^jparent for every 

program. 

Critical Element 1. Indicators should be tied to mission s t atements, goals and 
outcomes. Critical Element 2. The quai^ ' ty indicators should address the general 
areas of: recruitment, assessment (including needs), program planning and design^ 
retention, curriculum and instruction, program evaluation, staff development. 
su pport services and learning progress, gains and proficiency. 
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CORE COMPONENT 2. Projected program, process and learner outcomes should be 

established. 

Critical Element 1. Program outcomes sh ould include recruitmgnt. rgtgntipn. 
curriculum and instruction, staff develop ment and sunoort services, Prpges? 
outcc mes should include assessment, pro&r am planning and design, program 
evaluation. Learning outcomes should include, lea rning progress, gains, and 
proficiency . 

Critical Element 2. Several outcome areas will natu rally overlap, but key areas 
must be linked. For example, assessment mus t be tied to curriculum and 
instruction, program planning should be based on n eeds assessment and program 
evaluation, staff development should reflect content of curriculu m and instruction. 

Critical Element 3. Diversity of program partici pants must be reflected in 
pntcomes. A "one size fits all" outcome, such as "pa rticipants will complete their 
C^HD." is not appropriate for all adult ESL students. 

CORE COMPONENT 3. Outcomes must be defined with a standard of measurement. 

Critical Element 1 . Standards should provide a gau ge for measuring success SO 
that there is a distinct idea of what is an acceptable le vel of achievement for that 
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standard. For example, a common learner outco me in ESL programs is ''to learn 
English/' This outcome is difficult to measure. The ouicome must be more 
specific and must include some standard of achievement. An example of a 
vocational ESL standard might be "to develop proficiency in English and data 
entry in order to obtain employment (defined outcome") by accomplishing the 
objectives of an individualized instructional plan (standard of measurement^ 

Critical Element 2. Assessment tools should be compatible with the type of 
standard that is set. For instance, if the standard is "participants will be able to 
complete a job inter\iew proficiently in English." then it is not appropriate to use 
the BEST test as a measure of this standard because it does not test for 
proficiency in job interviews, for 
example. 

CORE COMPONENT 4. There must be a system for developing the indicators, 

outcomes and standards and it must be based on research, 
experimentation and trial and error. 

Critical Element 1 . Data and information must be collected that back up the 
system so that it is reasonable, realistic, valid and reliable. 
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Critical Element 2. The system must be ;<ftveloped from the too down and 
bottom up and must have suptx)rt and resources from both directions. 

Cri tical Element 3. Specific data for ESL populations n eeds to be collected. 

Critical Element 4. Evaluation and planning sho uld be an integral part of the 
system. 

CORE COMPONENT 5. A system of program monitoring and compliance and a 

policy regarding what will bappra if the program cannot 
demonstrate accountability, needs to be in place. 

Critical Element 1 . Accreditation systems for programs which have demonstrated 
accountability need to be a part of the system. 

Critical Element 2. Sanctions against programs that c annot demonstrate 
accountability and have been given adequate assistance and resources to do so. 
must be part of the system as well. 

Accountability: The Federal and State Response 
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It is not surprising that leaders, administrators and instructors of adult education 
have not gotten very far on establishing comprehensive models of accountability as 
described above because they are not in agreement about what should be the goals and 
outcomes of adult ESL, how it should be measured, and what importance should be 
placed on adult ESL programs in particular. Further, programs are so diverse, it seems 
like an overwhelming task to establish appropriate, feasible outcomes that are based on 
standards which can be measured. But even if more emphasis were placed on ESL in 
adult education or if there were a simple way to deal with the diversity, the topic of 
accountability would not be any farther developed than it is now. The reality is that no 
one in a leadership position is currently addressing the issue of accountability in a way 
that will make programs truly accountable for their actions, accomplishments and 
expenditure of funds. The federal government and some states ore getting there, but no 
one really has clear vision of what needs to be done. 

In 1991, with the passage of the National Literacy Act, came an opportunity to 
begin to establish systems of accountability for adult education, in general. Legislation 
was adopted that provided momentum for making improvements in adult education and 
making programs address some issues that are vital to demonstrating accountability. 
The National Literacy Act contains provisions that would enable the field to lay the 
necessary groundwork for a substantive program of accountability. The Act: 

1) opens adult education funds to all service providers; 
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2) increases teacher training funding allowances; 

3) requires states to develop indicators of program quality tc be used to 
evaluate programs funded through AEA (section 322, (c) (2); "to set forth 
measurable goals for improving literacy levels, reteniion in literacy 
programs and long term learning gains of individuals"(section 342, 
subsection (c) (2); to evaluate 20% of the grant recipients each year to 
determine the extent that the program has met the state's indicators for 
program quality; 

4) requires that the Secretary develop indicators of program quality to be 
used to judge success of programs in recruitment, retention and literacy 
skills of students; 

5) establishes an institute for literacy for conducting research, providing 
technical assistance and training, developing policy recommendations, 
developing quality indicators, and developing standards for program 
effectiveness; 

6) establishes literacy resource centers to help providers meet quality 
standards by improving programs, coordinating efforts and increasing the 
expertise of the field. 

While the legislation never mandates that the federal government or the state 
governments should initiate systems of accountability, it does have language that suggests 
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that programs 1) improve quality, 2) establish indicators of what that quality is, 3) 
identify standards for ensuring that quality has been met, 4) state measurable goals and 
5) identify what constitutes success. 

In July of 1992, the federal government offered its response to the legislation 
regarding the Secretary's requirement to develop indicators that can be used as models 
for the states in the document Model Indicators of Program Quali ty for Adult Education. 
Similarly, in July of 1993, the states submitted their required indicators that were 
approved by the U.S. Department of Education Office of Vocational and Adult 
Education. Using the above outline of core components and critical elements of 
accountability systems for adult ESL education, a comparison of what was adopted by 
both the federal government and the states to meet the requirements of the National 
Literacy Act legislation will help to determine what work still needs to be done to 
develop and ensure accountable programs. For the purposes of this paper, the five states 
that have the largest ESL populations are reviewed. They are: California, Texas, New 
York, Florida, Illinois. In addition, Massachusetts is examined because it has a 
significant number of ESL students and has included some unique features that warrant 
review. 

Core Component 1: Quality Program Indicators that a) are tied to mission statements, 
goals and outcomes; and b) address recruitment; assessment; needs; planning and design; 
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retention; curriculum and instruction; evaluation; staff development; support services; 
learning progress, gains and proficiency. 

In general, the federal government has taken a definitional approach to 
accountability, by describing what quality is and giving examples to illustrate the 
descriptions. It defines a quality indicator as "a variable that refiects effective and 
efficient program performance" (OVAE, 1992). Eight indicators covering program areas 
such as learner progress, program planning, curriculum and instruction, staff 
development, support services, recruitment, and retention are identified. While the 
indicators proposed are often broad e.g. "students remain in the program long enough 
to meet their educational needs;" (p.8) and sometimes vague, e.g. "learners demonstrate 
progress toward attainment of basic skills and competencies that support their educational 
needs;" (p.5), they cover the areas that current research holds to be essential for quality 
programs. In addition, each indicator is defined, to give the reader an idea of what is 
intended. 

Indicators seem to be tied to the goals of adult education, although there is littie 
specificity related to ESL instruction. Three indicators are related to outcomes, but they 
are very generally stated. 

Two vital areas that are not included in the federal government's indicators are 
assessment and evaluation. Evaluation is mentioned in one indicator, but not given 
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ample attention to support a comprehensive accountability system. Assessment is not 
identified as an indicator of quality. 

The states, as expected, have their own variations of the indicators of quality, but 
for the most part, have used the federal model as a basis. An exception is California, 
who had begun the work to establish indicators prior to the legislation. 

California's indicators are published in the document, Enplish-as-a-Second 
Language Model Standards for Adult Education Programs (1992), and strongly emphasize 
the instructional aspects of program development. The work of Ct ,fomia largely 
focuses on standardizing levels of achievement for adult ESL instruction. The intent of 
California's standards is to "help administrators and instructors organize ESL classes in 
an articulated sequence from ESL beginning literacy through advanced; assess and place 
students in appropriate levels of instruction; choose textbooks and teaching techniques; 
and evaluate the effectiveness of curricula and programs" (California Department Of 
Education (CDOE), 1992). Subsequently, California primarily addresses areas of 
program, curriculum, instruction, and student evaluation. 

Program standards include such items as sequencing courses, having a curriculum 
with learning objectives, and using multiple assessment measures for placement and 
promotion. Curriculum standards include focusing curriculum on meeting needs, having 
integrated instruction, and designing curriculum appropriately according to level 
Instructional standards cover areas of holistic language instruction, communicative 
competence, language acquisition and development, and instructional sequence and 
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activities. Student evaluation standards are tied to placement, monitoring progress and 
assessing proficiency. 

While progressive in three areas, California's standards seem to be narrowly 
focused on instructional design. They omit crucial categories such as staff development, 
recruitment, retention, support services and program evaluation. In addition, there is little 
indication of linking standards to program outcomes, but significant detail about how the 
standards should be implemented. Thus, California's framework suggests that quality 
is based on the process of instruction and assessment. Although somewhat incomplete, 
California's standards reflect the goals of adult education, and it is one of the few states 
that has focused specifically on the adult ESL population. 

Indicators of program quality for Texas are modeled after the federal 
government's work, but expand the areas significantly (Texas Education Agency (TEA), 
1993). A total of twenty-two indicators covering program planning, recruitment, 
curriculum and instruction, support services, professional development, and student 
retention are specified in the document. The indicators on learner outcomes are related 
to academic development, real wo/ld applications, preparation for transition, work force 
development, and personal development. Indicators in the program planning area include 
needs assessment and participatory, evaluation-based planning. Recruitment indicators 
consist of identifying needs and selecting appropriate strategies. Basing curriculum and 
instruction on adult learning theory, learner-centered and participatory approaches, 
functional contexts, and thinking and problem solving processes are outlined as important 
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indicators of quality as well as ensuring that curriculum changes according to evaluation 
information. Holistic assessment is identified as another indicator in quality programs. 
Support service indicators include matching services to needs, enabling students to access 
services, and working with the community to ensure full delivery of service. 
Professional development requires that collaborative planning is based on instructional 
proficiencies, needs and outcomes and emphasizes that staff development resources 
should be available to implement plans. The indicator on retention suggests that students 
should remain in the program to meet goals or make a successful transition to other 
educational programs. 

The Texas document clearly defines what each indicator means using a format that 
identifies the indicator followed by "this means that" statements which depict practical 
applications of the indicator. Overall, Texas does a thorough job of covering indicators 
that are representative of quality programs and the indicators are strongly tied to goals 
and outcomes. Knowing that developing quality indicators must be done with care and 
the appropriate involvement of representative groups. New York (State Education 
Department/The University of the State of New York (SED/USONY), 1993) opted to 
develop an interim set of three indicators in Phase I. A process for developing a 
comprehensive set of indicators under the guidance of a working group and a reaction 
group is part of Phase II. The New York plan also has a pilot test component, whereby 
providers will work with the indicators before the indicators become part of the system. 

New York's interim indicators for Phase I cover three areas: recruitment, 
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retention, and educational gains. The indicators are very broadly stated and allow for 
a great deal of flexibility. For example, indicator 3 states: "The program enables 
participants to demonstrate progress toward their goals." (SED/USONY, 1993, no page 
numbers given). The New York indicators generally relate to the goals of adult 
education, but are admittedly incomplete, and do not cover all areas that are cited to 
reflect quality in adult ESL education. Like most other states, they do not describe 
outcomes that should be achieved in the adult education programs. 

Florida has developed nine indicators of program quality that include: educational 
gains and outcomes, retention, recruitment, program orientation, program planning and 
evaluation, curriculum and instruction, support services, and staff development (Florida 
Department of Education (FDOE), 1993). The wording of Florida's indicators is almost 
identical to USDOE's with the exception of the indicator on retention, which is worded 
slightly differently to reflect that retention involves students entering, exiting and 
reentering programs as goals change. In addition, Florida has added an indicator on 
program orientation that addresses commitment to serving adults who need basic and 
functional literacy skills so they can contribute to the community. As with the federal 
indicators, there is a general indication of outcomes but program evaluation and 
assessment are not addressed. There is no definition of the indicators or other 
information to describe any processes associated with the indicators; they are merely 
inserted in the state plan as amendments. 
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Illinois adds the indicators of program coordination and institutional support to 
a framework which mirrors the USDOE model (Illinois State Board of Education (ISBE), 
1993). Wording is changed occasionally, but overall, the Illinois document closely 
resembles the model indicators of the federal government, and consequently, has the 
same problems. Two very important indicators are added: (1) provide direction for 
avoiding duplication of services to maximize resources and (2) ensure that program 
administration commits resources to meet quality requirements. 

Massachusetts has taken a unique approach to the development of its indicators 
(The Commonwealth of Massachusetts Department of Education (CMDOE), 1993). Its 
approach represents a shared responsibility for quality indicators between the state and 
local providers. Massachusetts indicators address six areas: state leadership, 
coordination, and accountability; program planning and community linkages; program 
and staff development; curriculum and instruction; development and implementation; 
adult learner participation; adult learner progress, achievement, and empowerment. 
Within each general indicator area there are several indicators, totaling twenty. 

The content of the Massachusetts' indicators is very different from the federal 
indicators. For state leadership, coordination and accountability, the first indicator is a 
mission-type statement, e.g. "...(ABE) services are an integral component of our state's 
strategies to support and strengthen our Commonwealth's families, communities and 
economy" (CMDOE, 1993, p.l). Three other indicators in this area are directed toward 
state officials: (1) being responsible for establishing coordinated services that are 



34 



accountable; (2) conducting planning that is ongoing, participatory, guided by evaluation, 
and based on regional needs and resources; (3) and ensuring that state level staff 
members are knowledgeable about ABE programs and can support them with appropriate 
technical assistance. 

indicators for program planning and community linkages cover areas related to: 
participatory planning based on evaluation information, needs and resources at all levels; 
adequate financial management and accountability; and appropriate formative and 
summative evaluation. Three indicators of quality are reported for program and staff 
development that relate to hiring, training and experience; service delivery; and program 
and staff development planning and implementation. Curricular and instructional 
indicators cover areas associated with learner progress and gains, participatory and 
learner-centered practices, and learner needs, goals and skill development. The area of 
adult learner participation includes support services, outreach and recruitment, retention 
and facilities. A final specific indicator is provided on learner progress, achievement and 
empowerment. 

Massachusetts is comprehensive in establishing indicators, but includes some 
confusing indicators. For example, the indicator on learner progress is quite difficult to 
follow; 

Improvement in Uie full range of skills which lead to increased learner self-esteem 
and empowerment, including linguistic, math, life skills, critical thinking and 
problem solving skills. Progress in these various areas is measured by means that 
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give meaningful information to adult learners and teachers. (CMSOE, 1993, p. 
15.) 

Another shorter indicator which is hard to interpret states that: "The program's physical 
facilities encourage rather than inhibit strong adult education practices." (CMSOE, 1993, 
p. 14). Massachusetts indicators, however, cover important areas, reflect the goals of 
adult education, and are tied to outcomes. 

Core Component 2: Program, process and learner outcomes should be established and 
linked where appropriate; outcomes should reflect diversity of populations served. 

The USDOE indicators include all three types of outcomes: program, process and 
learner. Quality indicators that contribute to the accomplishment of program outcomes 
include recruiting those who are in need of services; developing curriculum and 
instruction geared to student learning styles and levels of student needs; implementing 
staff training that is based on staft needs in terms of providing quality instruction; and 
providing support services that are based on needs assessment. Indicators that support 
process outcomes include: ongoing and participatory planning that is guided by 
evaluation, needs assessments and resources. Learner outcomes are related to retaining 
students long enough to meet educational goals; learners demonstrating "progress toward 
attainment of basic skills and competencies that support their educational needs" (OVAE, 
1992, p.5); and learners advancing in instructional programs or completing educational 
requirements so they can continue in educational programs or training. To sore 
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extent, the indicators proposed by the USDOE are linked in key areas such as gearing 
instruction, recruitment, and retention efforts to needs and basing planning on evaluation 
results. There is little indication, however, of important connections like designing 
curriculum and instruction to achieve learner outcomes or ensuring that assessment is tied 
to instruction. 

Because the quality indicators of the federal government are so general, it is 
possible to infer that there is some sensitivity to diversity. In addition, the description 
of the curriculum and instruction indicator provides clarification that there should be 
individualization in curriculum and instruction to meet the needs of various diverse 
populations. Nonetheless, this elaboration does not provide substantive direction 
regarding quality instruction for diverse populations; it is merely a recognition that 
individualization should exist. 

California has developed program and process outcomes, which are called 
standards, for ESL specifically. The outcomes are highly integrated and include 
extensive detail on how the curriculum should be developed, how instruction should be 
sequenced and designed, how assessment should be designed and used, and what 
instructional activities are appropriate. An example of an integrated program and process 
outcome follows: "The curriculum is focused on meeting students' needs as determined 
by assessments of students' language proficiencies, goals, and interests." (CDOE, 1992, 
p. 3). Learner outcomes are implied, but not overtly stated in the standards: for 
instance, •'Language tasks in the classroom consist of meaningful interchanges that 

37 



enhance students' communicative competence." (CDOE, 1992, p. 5). Finally, diversity 
in programming to meet participants' needs is reflected throughout California's document 
including a specific standard on providing instructional activities that integrate language 
and culture so students can compare American culture to their own. 

Even though California has done extensive work in establishing proficiencies for 
ESL students, the levels of proficiency established are discrete and discourage 
individualization and participatory involvement of the learner. Proficiencies established 
for certain levels do not totally reflect the unique language acquisition circumstances of 
the learner. For example, California's beginning level assumes that the learner has very 
little proficiency in reading, writing, speaking, listening or work related functions. 
There are many circumstances, however, when a learner might possess more advanced 
proficiency in reading and listening than speaking or writing. 

The Texas indicators cover outcomes for program, process, and learners; they are 
linked when appropriate and they reflect diversity. Specific learner outcomes exist for 
increasing proficiency in oral and written communication, problem-solving, and 
numeracy; improving capacity to participate as lifelong learners; demonstrating progress 
for attaining skills or credentials for accessing further education or training opportunities; 
demonstrating increased proficiency in academic skills related to work; and 
demonstrating increased proficiency in setting goals, self-assessing progress, and 
incorporating changes. Program outcomes are reflected in recruitment, curriculum and 
instruction, staff development and support service indicators. Program outcomes reveal 
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sensitivity to diversity, e.g. "Curricular and instructional processes reflect leamer-ce"tered 
and participatory approaches that are designed to meet individual learner needs." (TEA, 
1993, p.4). They are also linked to other appropriate areas, e.g. "curriculum and 
instruction are based on learner outcomes, are consistent with and supportive of adult 
learning theory, and are supported by research and knowledge of effective practice." 
(TEA, 1993, p.4). 

New York has one indicator on recruitment which could be considered a program 
outcome in that it states what the program should accomplish--"Recruits the population 
in the community in need of literacy services as approved in their application for 
funding." (SED/USONY, 1993, no page numbers given). The other two indicators reflect 
broad learner outcomes, "...participants stay in the program long enough to reach their 
goals," and "The program enables participants to demonstrate progress towards their 
goals." (SED/USONY, 1993, no page numbers given). The broadness of the indicators 
provides for diversity issues, but it is hard to say that New York is linking key areas at 
this time. 

The accomplishments and shortcomings regarding this core component for Florida 
and Illinois are similar to the federal model and no other important distinctions are worth 
noting. 

Massachusetts often combines program, process and learner outcomes in one 
indicator and covers all areas that are important to adult education. Further, 
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Massachusetts, links important areas together and their indicators reflect diversity. For 
example: 

The program has established processes and structures to ensure that 
curriculum and instruction provide for maximum feasible progress toward 
developing the skills foundation expected of a high school graduate, are 
empowering for adult learners and address their diverse educational and 
cultural backgrounds. (CMDOE, 1993, p. 11) 

The Massachusetts indicators are holistic in nature, making many of the indicators 
very dense with information. This may account for the lack of clarity mentioned earlier. 

Core Component 3: Outcomes must be defined with a standard of measurement that 
provides a gauge for knowing what an accq)table level of achievement is for that 
standard and assessment tools should be appropriately selected and used to measure the 
standard. 

The USDOE defines performance standards: 

A performance standard defines a level of acceptable performance in terms 
of a specific numeric criterion. For example, a retention performance 
standard specifies a minimum number of hours of instruction per student. 
A standard may be established over time or to measure changes in 
performance of time. (OVAE, 1992, p. 2.) 
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However, performance standards are not part of the model developed by the 
USDOE. Sample measures are given for each quality indicator to give providers an idea 
of what data might be used to determine the level of performar.ee achieved. The 
Department defines measures as "data used to determine the quantitative level of 
performance," (0VAE,1992, p. 2). Interestingly, the examples of measures provided 
are not always quantitative, especially for process or program type indicators. For 
example, program planning indicator measures suggest documentation like a planning 
document that specifies program goals and objectives, openness of the program to 
community input with evidence of advisory boards, and existence of program evaluation 
and a feedback system. No quantitative measures are mentioned for the program 
planning indicator. This is understandable because qualitative data is more appropriate 
for some process and program indicators. Nonetheless, it is unfortunate that the federal 
government's model presents a contradictory and confusing message iivthis regard. 

Another disappointing characteristic of the federal government's model mentioned 
above is that indicators are so general that all programs should be able to do something 
to address the indicator and there is no way to tell which programs are doing an adequate 
or good job and which are not. For example, the federal government's first indicator on 
education gains is: "Learners demonstrate progress toward attainment of basic skills and 
competencies that support educational needs" (OVAE, 1992, p. 5). To define this 
indicator, or give the reader some direction of what this means, the document offers the 
following: 
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Programs support learners' educational needs by promoting progress 
toward attainment of linguistic, mathematic, communication, and problem- 
solving competencies. Progress is demonstrated by improvement in 
participants' abilities to understand, speak, read and write English, 
perform basic computations, and function more effectively in the home, 
community and workplace. (OVAE, 1992, p. 5) 

Overall, this is a good indicator for everyone to be able to do something about, but 
examples of what constitutes progress are not apparent. Sample measures are given, 
such as standardized test score gains, competency test score gains, teacher reports of 
gains, and alternative assessments. However, there is no indication of what the extent 
of progress should be; no indication of what is reasonable improvement; and essentially, 
no standard for determining the degree of quality that should exist. 

Certainly most programs that receive AEA funds could easily show evidence of 
meeting this indicator, and may not work to improve because according to the federal 
government's model, the program is meeting quality standards if learners demonstrate 
any level of progress. Understandably, the federal government may not be willing to set 
forth the standards for demonstrating progress, or defining proficiency, because such 
actions have the tendency to be narrowly interpreted and may not allow for the flexibility 
that is required for local program needs. Plus, it is vital that local programs define 
progress and set standards because "grass roots" and "bottom up" participation in 

42 



accountability is necessary for a comprehensive system. But more specific guidance on 
establishing standards is appropriate, especially for adult ESL learners who come to 
programs with such diverse backgrounds and experiences. 

Finally, beyond examples of measures, there is littie discussion of assessment 
tools that should be used for measurement purposes. Assessment is a critical factor in 
determining outcomes, yet is seems to be grossly ignored in the federal government's 
document. 

Of course the states that adhered to the USDOE model, Florida and Illinois, face 
the same problem, as does New York, which has only three very general indicators at 
this time. Florida has an additional problem as well. It provides only the nine indicators 
and gives no definition of the meaning of the indicators or what measures might be used 
to document that they exist. Illinois identifies a specific measurement tool for ESL 
students that will be used to measure gains, the Combined English Language Skills 
Assessment (CELSA) test, but does not define standards or measures. 

Texas is more specific than the federal government or the other states, and has 
developed a qualitative system to try to establish local standards. Texas states the 
outcomes that should be achieved and what counts for quality programs, and statements 
that exemplify how to measure the outcome or indicator are provided. While the Texas 
document does not include standards per se, the m-^surement of achievement of a 
performance indicator happens in the context of results based monitoring. Local programs 
are required to verify existence of the indicators with specific data sources. As data 
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sources are reviewed, a qualitative measure (i.e., a benchmark) is derived for each 
indicator, and programs are expected to improve and demonstrate continuous progress. 
In the assessment of performance, local conditions are taken into account so that the 
assessment is site- and needs- specific. The standard is set in regard to each indicator 
and a determination of performance on each indicator is both qualitative and quantitative. 

Texas has developed adult education instructor and administrator proficiencies, 
which present some standards for instructional delivery and program administration. The 
draft documents include specific descriptions of what proficient instructors and 
administrators do, but no measures or performance standards are given. These 
documents do provide, however, comprehensive outlines of what Texas believes quality 
instructors and administrators need to be able to do. 

Massachusetts uses a somewhat different approach as those mentioned already by 
defining what the indicator looks like if it is in place, giving some idea of the measures 
that would be appropriate, and, in some cases, stating a standard of performance. Each 
indicator is followed by a description of what happens when the indicator is apparent in 
the program. For example, indicator 3.3 states that: "The program has established 
processes and structures to develop and implement an appropriate program and staff 
development plan." (CMDOE, 1993, p. 9). A suggested measure is, "Programs use the 
integrated program and staff development model developed by SABES." (CMDOE, 




1993, p. 9). The standard given states that all staff members attend the SABES 
orientation for fifteen hours. 

Generally, however, the Massachusetts model presents measures that are ver>' 
process oriented and seem more like extended definitions of quality indicators. 
Performance standards are not given for most of the indicators, but a few standards are 
included for staff activities such as student-staff ratios, time usage, and training. Support 
services have standards for counseling and recruitment which are either process or 
program based such as "The program dedicates not less that 3% of total staff hours to 
program development activities;" (CMDOE, 1993, p.8) or "Adequate staff hours are 
assigned to retention activities." (CMDOE, 1993, p. 13). In the second example, 
adequate is not defined. 

California's approach to standards relative to learner outcomes is novel. While 
California does not address program standards or process standards in much detail, they 
do describe elements of proficiency for six levels of ESL instruction. The levels indicate 
how instruction should be focused and illustrate what students should be able to do as a 
result of instruction. Exit criteria for listening, speaking, reading, writing, language 
functions, and language forms are listed, and consequently, provide the specificity for 
outcomes that i^ not apparent in the standards. Thus, at the end of the California ESL 
Advanced-High Level, a student would be able to "speak fluently in most formal and 
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informal conversations on practical and social topics, except when under tension or 
pressure" (CDOE, 1992, p. 37). 

California recognizes that assessment of their standards or outcomes is not simple, 
therefore a guide for ESL Testing Standards is provided in its document. The format of 
this section identifies nine general standards for testing such as using tests that are 
appropriate, reliable and valid for non-native speakers; interpreting scores on the basis 
of proficiency levels; using tests to gain information that is helpful to students; 
administering tests appropriately, standardizing scores; using multiple measures and 
testing opportunities; using appropriate content; and testing higher order thinking skills. 
Specific descriptions of what the assessment tools and procedures should look like are 
provided as well as guidelines for good assessment practice. Perhaps due to California's 
discrete model, the document does not include much discussion on authentic or portfolio 
assessment, which is useful in determining outcomes of ESL instruction. 

Overall, the California model is pioneering work for many aspects of ESL 
instruction, but it has shortcomings in terms of program issues such as how learners' 
individual progress can be measured in the system. Further, California has not addressed 
performance standards for all areas that contribute to quality ESL programs, such as 
recruitment, retention and staff development. 

Core Component 4, There is a top down and bottom up system for developing the 
indicators; outcomes and standards are based on research, trial and error, and 
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experimentation so that data is collected that support the system, especially for ESL 
populations; and the system is evaluated and improved. 

Essentially all states and the federal government tried to initiate the process of 
establishing accountability systems from top down and bottom up structures as legislated. 
When the USDOE contracted with Pelavin and Associates to help develop their indicators 
several tasks were completed to provide leadership to the states. The tasks included 1) 
reviewing what the states had already done relative to program quality; 2) reviewing what 
other related programs that support basic skills had done; 3) commissioning papers from 
experts in the field to examine issues related to program quality; 4) conducting focus 
sessions of representative groups in the field to obtain input on the indicators; 5) 
synthesizing the results of the focus sessions to develop core indicators which met criteria 
of consistency regarding the goals the AEA and NLA, while allowing for valid and 
reliable measurement, allowing for states' data collection capabilities, ability to aggregate 
data; 6) condiicting workshops for state director's review of the indicators; and 7) 
revising indicators after the review (OVAE, 1992). 

Some states have been more inclusive than others, and some states have taken 
more time working on their indicators than others. The extent of inclusiveness that was 
used by the states varies greatly. One state, Florida, had several meetings with only 
eleven task force members and the required hearings. While another state, 
Massachusetts, implemented a specific project. Greater Opportunities for Adult Learner 
Success (GOALS), involving over forty programs, 1,000 adult learners and state agency 
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staff (Massachusetts). Basically, all six states, except Illinois, went about the initial 
process of establishing indicators by appointing a representative task force, meeting 
throughout the year to review the indicators set forth by Washington, deliberating on the 
indicators that needed to be present in the state, and submitting them to the field for 
review at hearings. Illinois did not appoint a task force per se, but tried to get input at 
various meetings and discussions throughout the year. Some states had more of a 
collaborative effort in developing the indicators (California, Massachusetts, and Texas), 
but all states have indicated that their initial efforts are preliminary. 

Beyond the NIL program to fund efforts to establish models of accountability for 
five states and to conduct an accountability academy for the states (Alabama, Hawaii, 
Kentucky, New York and Tennessee), it is difficult to know where the federal 
government is going with their efforts to establish an accountability system. In addition, 
it is not possible to know at this time what kind of work will be done by the states who 
received the NIL grants, and if it will deal with issues that affect ESL programs. 

Obviously, the work of the federal government regarding quality, standards and 
accountability is incomplete. A publication on model indicators of program quality and 
performance measurement grants for five states is simply not enough to establish 
accountability for adult education. Furthermore, it is unclear what the next steps of the 
federal government will be. 
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State plans vary regarding their next steps in establishing accountability systems. 
Florida began a process for establishing standards and measures but put the project on 
hold until they see if the federal government will require something specific. California 
is working hard to gather information on appropriate assessment tools and publish the 
results. In addition, California is presently field testing their standards to see what is 
working and where there are gaps in the system. 

Texas assigned sub-committees to refine specific learner outcomes and to develop 
instructor and administrator proficiencies. Texas is trying to build credence in both a 
qualitative and quantitative assessment system, but there are no specific plans to develop 
state standards of performance or measurement. Current indicators are being field tested 
through a results based monitoring system that will be used for evaluating 20% of the 
adult education programs annually. 

New York's Phase 1 and Phase 2 process for developing the indicators is a 
natural and dynamic approach to develop an accountability system. During Phase 1, a 
volunteer group of 20% of AEA recipients are using the three indicators of retention, 
recruitment and learner gains in their program evaluations. New York is identifying 
measures for each indicator, collecting information on the measures, determining how 
the measures relate to program quality, and identifying ways to report the intormation. 
Some programs are also establishing outcomes for ESL as a voluntary experiment. The 
information gained from this pilot study will then be used to develop the statewide 
indicators and measures. For Phase 2 of the New York system, a working group and 
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reaction group will be appointed. The working group of 6 to 10 persons from the field 
with some expertise in performance indicators will develop a framework for the 
indicators, working with the agencies who participate in the 20% evaluation process. A 
reaction group of 15-30 individuals representative of the field will review and critique 
the indicators. Together they will develop a comprehensive list of indicators that will 
then be tested by 20% volunteer group of providers. This process continues annually 
until "all requirements of the 20% evaluation are complete" (SED/USONY, 1993, no 
page numbers). 

A plan for developing program outcomes or measuring them has not been initiated 
yet in New York, but more than likely the state will put more emphasis on learner 
outcomes and they are looking toward a system which has two categories: required 
outcomes that must be met and programs' individualized outcomes based on local needs. 
Regarding ESL, New York will have some different outcomes, especially in the areas 
of cultural and survival skills. They are developing different assessment procedures for 
this population as well. 

Illinois plans to develop performance standards this year with direction from the 
field. Local programs will be able to add their own indicators, but the actual process for 
developing standards has not been established yet. There is an indication that learner 
gains will be required as outcomes, but they have not decided how standards will be set 
as of yet. 
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Massachusetts will continue to work on their indicators throughout the year to 
improve them. They are trying to use research and data collection to determine what 
outcomes should be collected and what data can be collected on these outcomes, how to 
link goals to the outcomes and what is the linkage between such variables as progress and 
student participation. Massachusetts will try to develop a system of accountability that 
supports inputs and structures rather than just outcomes. Their systems will probably be 
"menu driven" to help deal with diversity issues, thereby allowing programs to decide 
on the outcomes that will be accomplished in accordance with needs. For ESL, 
assessment is a significant issue that must be addressed. Further, the idea of standards 
for ESL, when the needs and abilities are so diverse, is problematic. Consequently, 
Massachusetts is considering a goal based system of outcomes and standards for ESL to 
determine if students are making progress on their goals. 

Core Component 5: A system of monitoring and compliance which includes policies 
regarding endorsement of accountable programs and sanctions for non-accountable ones. 

The federal government and the states have not done much to ensure credibility 
regarding their systems of accountability and program quality. The USDOE does not 
address compliance issues, nor do many of the states. Even though there is a 
requirement for evaluating 20% of the programs that receive AEA funds, there is little 
indication of what will happen to those who cannot demonstrate that they have qualiiy 
programs according to the indicators. Florida is in the process of deciding how they will 
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monitor quality and is developing a new evaluation form that reflects the indicators. 
Information on California was not available. Illinois does not yet have plans for 
including the indicators in their evaluation process. New York has no formal guidelines, 
but they will give technical assistance to those who need it as will Massachusetts. 
Massachusetts has not decided what they will do yet either, but they are looking to see 
if programs reflect the quality indicators in their proposals and will fund proposals on 
the basis of their quality. They have also talked about the possibility of some type of 
accreditation system, but have made no formal plans as of yet. 

Texas will include the indicators in their Results Based Monitoring (RBM). RBM 
is "a single, integrated, comprehensive system to evaluate all programs," (TEA, 1993) 
which uses a state desk review of the annual performance report, application and 
complaints; a local study of the performance report, application, needs assessment, and 
student records; and site visits to verify data, review concerns; and development of a 
technical assistance program to ensure that program excellence indicators and compliance 
indicators are obtained. There is no indication of what will happen to programs that do 
not achieve results after receiving needed technical assistance. 

Problems With The Present Approach to Accountability 

The first and foremost problem of what has been done to date is that the field of 
adult education has not developed a system of accountability at the federal or state level. 
Bits and pieces of what should be in a system are present at all levels. Without question, 
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a lot of hard work has been done by many people dedicated to making programs 
accountable, and creative ideas and plans for developing systems at the state levels are 
in the works. The bottom line, however, is that a comprehensive system for genuinely 
establishing the accountability of programs has not resulted from the legislation of the 
National Literacy Act, and unfortunately it does not look as if the federal government or 
many states are very close to developing a system. 

It is possible to argue that the National Literacy Act legislation was never 
designed to ensure that the USDOE or the states receiving funds from AEA would 
develop systems of accountability. In fact, the AEA legislation is written so that the only 
requirement for states is to establish indicators of quality that relate to recruitment, 
retention and improvement of literacy skills and evaluate 20% of the programs each year 
to "consider the success of the grant recipient in meeting the State's indicators of 
program quality" (section 352 (2) (D). 

The Secretary is not exactly required to establish systems of accountability 
according to the legislation either. The AEA legislation reads, "the Secretary, in 
consultation with appropriate experts, educators, and administrators, shall develop 
indicators of program quality that may be used by State and local programs receiving 
assistance under this title as models by which to judge the success of such programs, 
including success in recruitment and retention of students and improvement in the literacy 
skills of students (Section 361 (c). This wording could be interpreted as a beginning 
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step for providing national leadership for accountability in adult education, but the work 
done through USDG i to meet this requirement went in a different direction. 

The approach taken by the USDOE was to simply describe the content of quality 
broadly in eight areas, and to recommend some general measures that could be used to 
demonstrate evidence of quality. The Department also provided some very general 
technical assistance through publications regarding what some states currently do for 
evaluating and monitoring programs, what other educational programs serving similar 
populations have done to demonstrate quality, and what some experts in the field felt was 
necessary for defining quality indicators. The result is a document with broad quality 
indicators that are mostly process oriented and that simply do not have much teeth to 
them. A program could easily document evidence of the indicators of quality presented 
by the federal government and still not meet the goals of adult education in a very 
effective way because the model has no standards for measuring performance or 
outcomes of programs. 

The lack of standards of performance and adequate measurements to accompany 
the indicators in order to verify quality is a problem that weakens the system. The 
concept of performance standard is defined, and an example outlined, but no direction 
on how to determine performance standards or what degree of measurement should be 
involved is provided. Requiring that programs identify indicators of quality, but not 
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requiring them to measure whether or not they can demonstrate quality is a flaw in the 
present approach. 

Finally, omitting guidelines and criteria for systematically monitoring programs 
to determine how they have implemented the quality indicators, is problematic as well. 
Except for the already existing requirement that 20% of the programs will be evaluated, 
no real mechanism has been established to determine that states and programs did what 
they said they would do. In fact, the federal government graciously avoided this 
important component of accountability. The reasons for the shortcomings in the 
federal approach to accountability may not be the fault of the USDOE, however. The 
National Literacy Act diffuses the responsibility of establishing accountability for 
programs by giving different parts of the job to different entities. For example, the task 
of establishing the indicators is accomplished by the Secretary and the states, but the task 
of recommending performance measures and standards for program effectiveness is to 
be done by the NIL, Some crucial work has yet to be done because the Institute has 
gotten off to such a slow start, but the real problem is the distribution of the 
responsibility. The relationship and role of the Institute to the Department and the states 
is peripheral with little authority to govern the development of an accountability system. 

As a consequence of the problems at the federal level, many states are making 
some of the same errors, Florida has adopted the federal system almost entirely and in 
fact has decided not to proceed with establishing measures and performance standards 
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until they see what the USDOE will do next. Illinois is in a similar boat, although they 
do have plans to develop standards of performance this next year. 

California of course has gone beyond what any state has accomplished for ESL 
by establishing mcxiel standards and defining proficiency levels for second language 
learners. This work is critical as it provides a framework for ESL outcomes that reflects 
holistic instruction. California has also begun to critically look at ESL assessment issues 
which are crucial to appropriately measuring standards. An accountability system cannot 
function if the assessment tools do not accurately measure performance or outcomes. But 
California has only dealt with one area of what comprises quality programs, curriculum 
and instruction. It has not dealt with program and process outcomes related to 
recruitment, retention, staff development, planning and needs assessment. 

Texas' list of indicators is comprehensive and reflects what good programs 
should look like according to the literature on quality. The state also developed a list of 
learner outcomes that are tied to curriculum and instruction, and have developed 
instructor and administrator proficiencies. Texas has developed a system for determining 
and monitoring the implementation of indicators, but it has not identified what is an 
acceptable level of performance. The RBM system of Texas appears to be a promising 
model for examining what programs have accomplished, but the state has not addressed 
compliance issues or how the model might be used for accreditation of programs. 

New York's Phase I and II system of developing the indicators is a viable action 
research type process that could serve as a good vehicle for collecting data to establish 



performance standards and field testing the system. New York's approach also is strong 
in terms of balancing the top down and bottom up input needed to develop an 
accountability system. The jury is not out on what type of standards they will come up 
with, however, plus their indicators are so broad that ?.t seems anything can count as 
quality. 

Massachusetts developed a valuable model for balancing agency and provider 
responsibility in the accountability process. In addition, Massachusetts considers a 
program's use of resources, which is not apparent in most other state's systems. 
Massachusetts' indicators are mostly process oriented, but they have established some 
standards of performance. The state is also developing a data collection system to 
conduct research for establishing other outcome standards which is important for the 
development of a valid and reliable system. 

On the whole, the federal government and the states have been able to solve some 
pieces of the accountability puzzle, but there is still work to be done at all levels of adult 
ESL education. Following are some specific recommendations to help move the field 
forward in establishing a comprehensive system of accountability for adult ESL 
education. 

Recommendations for Developing Accountability Systems for Adult ESL Education 

Because of the diversity of adult ESL education programs, which serve language 
minority students who rarely enter programs with homogeneous characteristics, the task 
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of establishing systems of accountability is more than overwhelming. But it must be 
done, and it must be done carefully, thoughtfully and with adequate resources. 

In order to expand on the work that has already been accomplished, four key 
policy areas need to be addressed: 1) legislation 2) leadership 3) research and technical 
assistance and 4) focus on adult ESL education. Each area is discussed below with 
specific policy recommendations. 

Legislation 

If adult ESL education programs are going to demonstrate accountability, it 
appears that the legislation of the National Literacy Act will have to be revisited. 

• Greater specificity regarding what the federal government and states need 
to do in order to establish accountability for adult ESL education must be 
written into the legislation. 

• Adequate resources to establish accountability systems must accompany 
the legislation. This responsibility cannot be accomplished within the 
present underfunded system of adult education. 

• The legislation has to have clear direction on what is required for states 
to develop systems of accountability appropriate for adult ESL education. 
This means that the legislation must be more specific regarding 
requirements on standards, outcouies, measurements, monitoring and 
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compliance. It does not mean, however, that the legislation has to reHect 
a national testing or standards system because a "one size fits all" 
approach to accountability for adult ESL education is simply inappropriate 
and contrary to the characteristics of meeting needs of diverse learner 
populations. States and local decision makers must be responsible for 
developing their own customized accountability systems. 
• The legislation should reflect an approach to accountability that supports 
the notion that standards, outcomes, measurements, monitoring and 
compliance issues can and should be negotiated from the bottom up to 
reflect participatory and learner-centered practice; but should also be 
based on core components and critical elements of comprehensive 
accountability systems that are supported from the top down. 

Leadership 

In order to demonstrate accountability for adult ESL education, there must 
national, state and local leadership for planning, developing and implementing 
appropriate systems. 

• The USDOE has 50 documents that specify the state plans for quality 
indicators. Those plans must be carefully reviewed and critiqued to 
determine which states are developing systems of accountability that show 
promising practices and can be used for replication. 
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• The NIL needs to support more widespread development of accountability 
systems. Funding five states to develop performance measurement 
systems, even though they are to share their results, seems very limited. 
The NIL must be more inclusive and use some of its funding to establish 
a working group for adult ESL education that can develop a framework 
for approaching accountability more systematically. 

• The NIL working group should have membership from at least seven 
states which serve the largest ESL populations, agency representatives that 
have a vested interest in adult ESL, local providers, adult ESL learners 
from representative language minorities, and adult ESL and accountability 
experts. The working group should be charged to thoroughly research and 
evaluate existing education accountability systems including what is 
presently happening in the states, recommend an appropriate system for 
adult ESL education, construct a plan for developing the system, and field 
test the system at state and local levels. 



Research and Technical Assistance. 

Funding must be provided to develop the needed research base for establishing 
standards of performance and appropriate measures for adult ESL education. 

• A National Center for Research in Adult ESL Education needs to be 
established that has the following functions: 
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1) 



conduct needed research for adult ESL accountability 



2) 



establish a data base for the development of performance 



standards, 



3) 



develop appropriate assessment instruments for measuring 



performance standards 

4) provide technical assistance in developing accountability systems 
for Adult ESL providers 

5) disseminate results. 

• States must establish programs of technical assistance to help providers 
develop quality standards and accountability systems, connected with 
resource centers, and based on models such as the Texas Results-Based 
Monitoring System, or the New York action research plan. 

Focus on Adult ESL Education 

An office of adult ESL in the USDOE must be established to look after concerns 
of this special population. 



• Issues of language minority adult populations must be adequately 
addressed by the federal government. A system of accountability is a 
critical component of generating quality ESL programs, and such a system 
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cannot be fully developed until there are decision makers in the federal 
government who have the responsibility to see that it happens. 
• Specific research and development funds need to be appropriated for this 
office to carry out a national agenda for adult ESL education. The United 
Kingdom and Australia have developed national frameworks for adult ESL 
accountability, and their work needs to be examined carefully for 
implications in the United States. 

Conclusion 

With dwindling federal dollars and public pressure to eliminate programs that are 
not effective, adult education has to show its worth. A "trust us" approach to 
accountability will not suffice. Product-oriented outcomes, measurement systems to 
determine if the outcomes are achieved, evaluation systems to make adjustments for 
program improvement and external monitoring systems to verify achievements must be 
implemented. But not at the expense of important and necessary process outcomes. 
There must be a balance, especially for adult ESL education. 

The eleventh hour is near and it is time to get very serious about the specific 
achievements and outcomes of adult ESL education. It is true that programs are 
underfunded and more resources are needed. But it is crucial to demonstrate the return 
on investment that taxpayers and learner^ are getting for their money and effort. Adult 
ESL education must take on this very difficult challenge and establish a viable system of 
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accountability that meets the unique and diverse needs of adult learners. The challenge 
must be mt by adult educators, or it will be taken on by others outside of the field. 
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